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(57) Abstract 

The apparatus (10) is a color wide dynamic range apparatus which includes a filter (12) interposed immediately in front of reoccurring 
color elements so that each pixel represents a given color element for the scene. At least two exposure levels are taken of the scene and 
the pixel outputs are decoded to generate the video luminance and chrominance signals. The images of the at least two exposure levels ate 
combined to form a final image. 
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• COLOR WIDE DYNAMIC RANGE CAMERA 

nflTKC A CHARGE COUPLED DEVICE WITH MOSAIC FILTER 

1* 



CROSS-REFERENCE TO RELATED APPLICATIONS 
This application is a continuation-in-part of U.S. 
patent application Serial No. 07/795,350 filed November 20, 
1991 entitled "Color Wide Dynamic Range Camera", which is, 
5 in turn, a continuation-in-part of U.S. patent application 
Serial No. 07/388,547, filed August 23, 1989, now U.S. 
Patent No. 5,114,442. Additionally, this application is 
related to U.S. Patent No. 4,858,014 and currently pending 
U.S. patent application Serial No. 07/805,512, filed 
10 December 11, 1991. The disclosures of all of the above- 
. identified U.S. patents and patent applications are 
incorporated herein by reference. 

BACKGROUND OF THE INVENTION 

15 Field of the Invention 

This invention pertains to video imagery and more 
particularly to apparatuses and techniques for providing 
enhancement of video color images. In particular, the 
present invention uses a four-color mosaic filter with a 
20 single chip CCD in conjunction with color wide dynamic 
range algorithms. It is also applicable, however, to other 
types of mosaic filters known in the art. 
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Description of the Prior Art. 

Various types of video enhancement apparatuses and 
techniques have been proposed. Prior implementations of 
color wide dynamic range cameras, such as those disclosed 
5 in the above-identified parent applications hereto, have 
used a plurality of CCD chips to generate the image data 
for subsequent processing. The use of multiple CCD chips, 
however, adds to the complexity and cost of the instrument. 
Moreover, current consumer video cameras, i.e., camcorders, 
10 almost universally use a single CCD chip. Therefore, a 
single CCD implementation is required to use dynamic range 
enhancement algorithms in a camcorder. Single CCD chip 
implementations are similarly preferred for endoscopic 
applications . 

15 

OBJECTS AKD SUMMTOIY OF THE INVENTION 
It is therefore an object of this invention to provide 
a color wide dynamic range camera implemented with a single 
CCD chip. 

20 It is also an object of this invention to provide a 

color wide dynamic range camera which is adapted for use 
with a camcorder. 

It is a further object of this invention to provide a 
color wide dynamic range camera which is adapted for use 
25 with a conventional endoscope. 

These and other objects of the invention will be more 
apparent from the discussion below. 

SUMMARY OF THE INVENTION 

30 There is thus provided in accordance with the 

preferred embodiment of the present invention, video 
imaging apparatus including means for providing a plurality 
of video color images of a scene at different exposure 
levels using a single CCD chip, each color image being 

35 separated into several (e.g., four in the preferred 
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embodiment) different components prior to sensing by the 
CCD chip by way of a multiple color mosaic filter in front 
of the CCD chip. The pixel outputs are then decoded — 
subjected to specific mathematical operations by the 
5 processing electronics following the CCD output — to 
generate the video luminance and chrominance signals. 

The present invention integrates the digital 
processing of the mosaic color CCD data witti ADAPTIVE 
SENSITIVITY™ dynamic range enhancement. This integration 
10 provides for a substantial savings in total' system 
processing hardware chip count and cost. It also permits 
better control of the color and detail production of the 
camera's video output. The mosaic storage format also 
provides for a unique video image compression technique. 

15 

BRIEF DESCRIPTION OF THE DRAWINGS 
Further objects and advantages of the invention will 
become apparent from the following description and claims, 
and from the accompanying drawings, wherein: 
20 Figure 1 is a general block diagram of ttie present 

invention - 

Figure 2 is a representative illustration of the data 
image elements, with the size of the data image elements 
exaggerated. 

25 Figure 3 is a general block diagram of th^e long and 

short processing of the present invention. 

Figure 4 is a block diagram of the color path of the 
present invention. 

Figure 5 is a block diagram of the intensity path of 
3 0 the present invention. 

Figure 6 is a block diagram of the looK-up table 
processing of the present invention. 

Figure 7 is a block diagram of the joint operations of 
the present invention. 
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Figure 8 is a block diagram of the differential color, 
intensity result block of the present invention. * 

Figure 9 is a block diagram of the color suppression 
factor block of the present invention. r 
5 Figure 10 is a block diagram of the color conversion 

block of the present invention. 

Figure 11 is a block diagram of the mosaic generation 
block of the present invention. 

10 DETAILED DESCRIPTION OF THE PREFERRED EMBODTMENTS 

Referring now to the drawings in detail wherein like 
numerals indicate like elements throughout the several 
views, one sees that Figure 1 is a block diagram of the 
apparatus 10 of the present invention. 

15 Apparatus 10 includes a mosaic filter 12 which is 

bonded to the front of CCD 14 (preferably a sixigle chip) , 
generally as part of the CCD integrated circuit 
manufacturing process. 

The alternating mosaic filter elements are cyan, 

20 magenta, yellow and green (wherein C 'cyan' - G 'green' + 
B 'blue'; M 'magenta' = R 'red' + B; and Ye 'yellow' = 
R+G) . When the CCD 14 charge output is rea<i out, the 
photoelectric charges from vertically adjacent sensor 
elements of CCD 14 are combined in the an^tlog shift 

25 register (not shown) . The on-chip addition gives rise to 
a, )3, 7 and 6 elements as described below and as described 
in the Sony CCD 1992 Data Book (Sony part number ICX038AK) , 
as well as earlier editions. 

As shown in Figure 2, the mosaic complementary 

30 additive color image comprises alternating first and second 
rows of image data elements 18^ — 18d wherein first rows 
include alternating a and 7 data elements (18a and 183, 
respectively) , and wherein the second rows include 
alternating jS and S data elements (18c ^^or 

35 respectively) . 
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The a image data elements 18^ are an ecjual mixture of 
cyan plus green (i.e., C + G = B 'b:j.ue' + 2G green'). The 
y image data elements 18b are an equal mixtu^re of magenta 
plus yellow (i.e., M + Ye = 2R 'red' + B + G) . Similarly, 
5 the jS image data elements 18c an equal mixture of cyan 
plus magenta (i.e., C + M=2B + G + R) and the 6 image 
data elements I8i, are an equal mixture of green plus yellow 
(i.e. , G + Ye = 2G + R) . 

Those skilled in the art will recognize that: 

Y (i.e., intensity) = = 

from which the definition of intensity (Y) in the red, 
15 green, blue (RGB) system may be derived: 

Y = R + 1.5G + B 

Of course, those skilled in the art will realize that 
other color combinations are equally applicable. Each 
mosaic element of filter 12 covers the sum of two adjacent 

20 pixel sensors of CCD 14 so that each pixel output^of CCD 14 
is representative of one of the above color combinations 
given for the various image data elements 18. Four 
different monochromatic images, each representative of one 
color combination chosen from the colors of image data 

25 elements 18^ - 18d of a given scene, are therefore generated 
by CCD 14. 

As can be further seen from Figure 1, apparatus 10 
includes four major functions as summarized below: 

1. Long/short exposure processing: 
30 The first stages of the algorithm are performed on the 

two exposures (long/short) separately. The processing 
of each exposure is divided into two paths: 

a. Color path processing - evaluates color component 
for each pixel. 

35 

b. Intensity (Y) path processing handles intensity 
information for each pixel. This includes point 
("DC") intensity and edge information. 
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Point processing: 
Each of the long/short exposure length processing 
functions (typically implemented on separate chips) 
outputs its point intensity information, obtained from 
the Y path processing, to four look-up tables (LUTs) . 
These tables determine the point intensity result of 
the two exposures, the normalized color weighting or 
color selection function and the saturation color 
suppression factor • This information serves the joint 
operation processing stage. The four LUTs are 
programmable, thus enabling operation with different 
functions when necessary. In an alternative embodi- 
ment, these LUTs may be replaced by a programmable, 
piecewise linear (PWL) or other digital function 
generator. 

Joint operations processing: 
Joint operations processing joins results produced by 
the long and short processing blocks, and results 
obtained from the functions implemented in the table 
processing LUTs, and evaluates the final output of the 
algorithm. The processing is divided into: 

a. Color components and Y result calculation — 
evaluates the final result of the color 
components and the intensity of each pixel. 

b. Color suppression factor calculation — evaluates 
the color suppression factor for each pixel, 
based on both edges and saturation information. 

c. Color conversion processing — converts mosaic 
differential color space to RGB color space and 
produces RGB and Y/Cr/Cb outputs for each pixel. 

Generate mosaic processing. 

Generate Mosaic processing converts RGB color space 

back to mosaic color space for each pixel. The mosaic 

information generated enables economical hardware 

storage of processed images. This information can be 

retrieved and replayed through the algorithm — in 



SUBSTITUTE SHEET (RULE 26) 



wo 94/18801 



PCT/US94/01358 



7 

Replay Mode — to produce RGB or y/cr/Cb output of the 
stored result. 

Referring now to Figure 1, similar to U.S. Patent No. 
5,144,442 and parent U.S. patent application Serial No. 
5 07/795,350 (the disclosures of which, again, along with 
U.S. patent application Serial No. 07/805,512 and U.S. 
Patent No. 4,858,014 are incorporated herein by reference), 
apparatus 10 includes long/short processing as implemented 
by mosaic long exposure field block 20 and mosaic short 
10 exposure field block 22 which obtain, respectively, a long 
and a short exposure from CCD 14 in order to allow subse*- 
quent processing by long exposure pr-ocessing block 24 and 
short exposure processing block 26. The terms "long" and 
"short" exposures are used here generally to denote two 

15 image inputs to apparatus 10. In general, "long" is used 
to mean an input with a higher exposure level, and "short", 
a lower exposure level. The higtier exposure may be 
generated in several ways, including longer integration 
time, typically obtained by controlling the "electronic 

20 shutter" of the CCD chip; higher gain in the analog 
amplifiers preceding digitization; oar a larger mechanical 
iris opening or other external gatincr means. 

These two image inputs are usually generated by a 
single CCD chip, but may also be generated simultaneously 

25 by two separate, boresighted CCD chips, as disclosed in the 
aforementioned earlier applications. For the more common 
case in which the two inputs are genearated by a single CCD 
chip, they may be generated either secjuentially (as in the 
case of the first method above — integration time control) 

3 0 or concurrently (by using two input channels with different 
gain levels) . When a sequential method is used, field 
memories are required at the input to apparatus 10 (in 
blocks 20 and 22) to synchronize the data coming from the 
two sequential fields or frames. These memories are not 

35 needed in concurrent modes, except for purposes of 
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"freezing" the image for electronic, digital storage. 
Switching logic incorporated in blocks 20 and 22 controls 
the data flow into and out of these field memories, 
depending on which mode (sequential or concurrent) is used. 
5 Of course, this implementation could be expanded to more 
than two exposure levels. Blocks 24 and 26 may typically 
be provided on separate processing chips or incorporated 
together in a single chip. The processing for each 
exposure is divided into two paths: 
10 1. Color path processing — handles color information 

for each pixel (see color path block 28 in Figure 
3 and, in more detail, in Figure 4) ; and 
2. Intensity (Y) path processing — handles intensity 
information for each pixel (see Y path block 30 
15 in Figure 3 and, in more detail, in Figure 5). 

Additionally, as shown in more detail in Figur-e 3, 
long/short exposure processing blocks 24, 26 include mosaic 
white balance block 32. 

Mosaic white balance block 32 receives the following 
20 field of information from long/ short exposure field blocks 
20, 22: 

a 7 a 7 a 7 • . • . 

0 6^606 

a 7 a 7 a 7 .... 

25 0 S P 6 P 6 

a 7 a 7 a 7 ... . 

0 S 0 6 0 6 

• . • 

30 ... 

That is, the information from CCD 14 with the mosaic 
data order intact is received. 

After processing, the mosaic white balance blocK 32 
outputs color-corrected data values: 
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wo 94/18801 



PCT/US94/01358 



9 





7wb 


^wb 


7wb 


ttwb 


7wb • • 


^wb 


^wb 


Pwb 


^wb 


^wb 


^wb • • 


ttwb 


7wb 


«wb 


7wb 


^^wb 


7wb — 


^wb 


*wb 


iSwb 


^wb 


^wb 


^wb • • 


ttwb 


7wb 


^wb 


7wb 


ttwb 


7wb • • 


^wb 


^wb 


^wb 




^wb 


^wb • • 



• • • 



10 Mosaic white balance block 32 contains mosaic color 

balance functions. These functions may typically be 
implemented as eight mosaic white balance LUTs C^oolt-up 
tables) . That is, for each exposure there is a set of four 
LUTs, one for each mosaic data type: a, 0, y, and S. 

15 Independent calculation of white balance correction factors 
is performed for each exposure. This enables white 
balancing scenes where the observable parts of the two 
exposures are at different color temperatures. Tlie LUTs 
may contain multiplicative correction factors which are 

20 evaluated as follows: 

From the definitions of a, 0, y, 6 and Y it follows, 
that for a white image (where by definition R^G=B) , the 
following relations should hold: 



25 




7 * 






= f.v 


30 


7wh 




35 




= f.v 



SUBSTITUTE SHEET (RULE 26) 



wo 94/18801 



PCTAJS94/013S8 



45 



50 



10 



Based on these relations correction factors can be 
calculated by enforcing these relations on the average of 
a white image: 

f * Y 



10 S * Y 



15 

_7 1. 



y ^ 

20 * ? 



25 where Y denotes a selective average over Y in the given 
white image and a, 'p, y and 6 are the respective average 
values of a, ^, y and Saturated or cutoff pixels are 

excluded from this average. Since by definition, 

30 Y= ^^ = ^-^ 

the equations for the correction factors aire: 



c . f . 1 

> 7 I 7 



c, = ^ * 



The LUT values are calculated by simple multiplication of 
each mosaic data type by its respective correction factor: 
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a 


* 










* 






Twb = 


y 


* 




5 




6 







In an alternative eiabodiment, these LUTs are replaced 
by digital multipliers. Furthermore, the LUTs may also be 
loaded with correction functions other than simple linear 
multiplicative factors. Alternatively, the mosaic balance 

10 correction factors can be computed based on four average 
signals, namely, a, )3, 7 and S, instead of merely two of 
them as above. This alternative yields improved laniformity 
of color balance under difficult conditions. Alterna- 
tively, the white balance function may be done on the RGB 

15 color components in the color conversion block 78 
(described below) . ^ 

Referring now to Figure 4, color path block 28 is 
shown in more detail. As previously stated, the input to 
color path block 28 iS; the image 'data d„^, fi^^, y^^, after 

20 processing by mosaic white balance block 32. 

The initia^ ;^;i^ceS;S^thg of color path block 28 is 
performed by color difference evaluation block 34 which 
receives data a^,,, j3^, 7^1,, from mosaic white balance 
block 32 and calculates color difference components dr, db 

25 for each pixel in the array: 

dr dr dr . . . . - (7-a) (7-a) (7-a) .... 

db db db . . . . = (5-/8) (S-p) (6-^) .... 

dr dr dr ♦ . . . = (7-a) (7-a) (7-a) .... 

db db db . . . . = (5-)3) (8-0) (6-0) .... 

30 wherein: 

dr = 7 - a , the differences between successive 
readings in even lines. 
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db s 5 - jS , the differences between successive 
readings in odd lines • 

The correct evaluation of dr and db requires 
horizontal interpolation as described in the following 
5 equations: 

For each pixel (i, j) in even lines i: 

OE^b(jodd-l) + a^(3odd+l) 

dJ^(jo(id)-7wb( jodd) 2 

10 ^ V 7wb(jcv «»"l) + Twb(jc.eo+l) 

<ir{3eveo)= 2 Ofwb(3cveo) 

where j is the pixel index along the line (here and 
henceforth) . 
15 For each pixel (i, j) in odd lines i: 

)8wb(jodd-l) + /3^{jodd+l) 
dh( jodd) 2 

20 5wb(jevco-l) + ^wb(jev«+l) ^ , 

db(3„„)= g Pwb(Drv«) 

Color difference components dr, db are thereafter 

25 received by low-pass color component bloclc 36 which 
calculates a low-pass color component drjp or db,p for each 
pixel: 

dr^dr,pdr^, . • . • = (y-OL)^ ly-<^)ip • • • • 

db,pdb^db|p . • . . = (5-i8)ip . . . . 

30 drjpdrjpdr^, , • . • = (7-a)ip (7-^^)^ . . . • 

db^db^,db,p . . • • = (^-^),p (^-/S)ip . - • • 

Block 36 performs horizontal low-pass filtering on dr^p 
and dbbp calculated in block 34. This reduces color 
artifacts caused by interpolation. In a preferred 
35 embodiment, the low-pass filter width is five pixels and 
its coefficients are Vk, h, h, h, ^A. The equations follow: 
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For pixels (i,j) in even lines i: 

dr(j-2) + 2*dr(j-l) + 2*dr(j) + 2*dr(j+l) + dr(j+2) 
dr^(j)- — 



For pixels (i,j) in odd lines i: 

db(j-2) + 2*db(j-l) + 2*db(j) + 2*db(j+l) + db(j+2) 



db^(j)- 



10 



Delay buffer 38 receives the output from low-pass 
color component block 36 and directs db^(i,^„ -1) , &t^{L^ - 
i) f db^,(icveo "*"i) and dr^(iodd+i) to vertical interpolation 
block 40 and dr(i„„) and db(i^) to multiplexer 42. 
15 Vertical interpolation block 40 receives tlie low-pass 

color components as described above and generates 
interpolated low-pass color components dr'ip in the odd 
nxmbered lines and db',p in the even numbered lines: 

db'^ db'^ db'^ db; 

20 dr'^, dr'^, dr'^ dr^ 

db'^ db'^, db; db; 

dr; dr; dr; dr; 

The equations follow: 
For even lines i: 
25 db^(i-l,j) + db^(i+l, j) 

db;(i, j)= = 



For odd lines i: 

dr^(i-l,j) + dr^(i-fi, j) 



30 cir;(i,j) = 
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The interpolated low-pass color components dr',p, db'^, 
are multiplexed with the original low-pass components dr,p, 
db^, to give the color path output values dr and dp for each 
pixel. This function is performed by multiplexer 42, which 
5 separates the output received from delay buffer block 38 
and vertical interpolator block 40 into a first path 
including db^' (i^^) and dbip(i^) and a second path including 
ctr^ (iweo) and dr',p(i^) . 

Referring now to Figure 5, which discloses in more 

10 detail the intensity (Y) processing block 30 shown in 
Figure 3, one sees that the input to intensity (Y) 
processing block 30 from mosaic white balance block 32 
(Figure 3) is received by intensity evaluation block 44 
which outputs computed intensity Y for each pixel - 

15 since only one of the four data types (a, 0, y, 6) is 

present at any given pixel, the intensity evaluation block 
44 calculation is performed as follows (based on the prior 
definition of Y) : 



20 



For pixels (i,j) in even lines i: 

Y(j^) = y^^^ : 



) = « (3) + 7(3-1) 
For pixels (i,j) in odd lines i: 

* V J even f 2 



The output from intensity evaluation block 44 is 
35 received by delay buffer 46, generate output intensity 
block 48 and limit block 50. 

Delay buffer 46 is a delay line of two horizontal 
lines, required for the 3x3 and 1x3 matrix transformations 
in Y path block 30. Together with the color path delay 
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buffer 38 and with Y path delay buffer 54 , it may be 
implemented in a preferred embodiment in mosaic data space, 
operating on the input a, 0, y, 6 data before the intensity 
(Y) evaluation block 44 and color difference evaluation 
5 block 34. It is shown here schematically for clarity. 

Vertical low-pass filter 52 receives intensity (Y) 
signals from the intensity evaluation block 44 as delayed 
by delay buffer 46* Block 52 generates the vertical low- 
pass intensity Y^^, defined as: 

10 Y,^(i,j) = Y(i-l.j) + 2Ya,j) + Y(i-fl,j) 

The unfiltered intensity (Y) input will sometimes 
exhibit horizontal stripes, one pixel high in each field, 

15 in areas of transition to saturation. These stripes stem 
from the different color spectra of the a, )8, 7, and 6 
pixels, as a result of which the a+7 value of Y{i(even),j) 
may, for instance, reach saturation at a lower level of 
optical intensity than the )3+5 value of the vertically 

20 adjacent Y(i+l(odd) , j) . Yvip averages these values to obtain 
a function that is smooth over the transition area. 

Generate output intensity block 48 receives intensity 
(Y) information from intensity evaluation block 44 and 
vertical low-pass intensity (Y^^,) information from vertical 

25 low-pass filter 52. The output of block 48 is output 
intensity (Y^ut) to point processing LUT block 62 (see Figure 
!)• 

Block 48 replaces the original luminance Y, computed 
by the intensity evaluation block 44, with Y^^ when Y^jp 
30 approaches saturation, in order to prevent the appearance 
of horizontal stripes as explained above. Block 48 
implements the function: 
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The value of Yto^ow is typically equal to approximately 
5 220 on an 8-bit scale of 0-255. As values of Y approach 
saturation, image detail is lost ±n any event, so that 
substituting Y,^ in the high range does not adversely affect 
the perceived resolution. Y,^ is used as the selecting 
input in order to ensure a smooth transition. 

10 Limit block 50 receives intensity (Y) signals from 

intensity evaluation block 44 and generates limited 
liiminance Limit block 50 cuts off the upper range of 

intensity (Y) values that are to be Input to edge detection 
block 56, in order to prevent deteci:ion of false edges or 

15 horizontal stripes that can arise in areas of transition to 
saturation. Limit block 50 implements the function: 

Yu^ = min {Y, Yu„,} 

the value of is typically equal to approximately 220. 

The output of limit block 50 (i.e., Y^^) is delayed by 
20 delay buffer 54 and received by ed^e detection block 56 
which outputs edge information for each pixel. 

Edge detector block 56 convolves the Yi^^ value and its 
8 immediate neighbors, with a high-pass or edge detecting 
kernel . 

25 In one embodiment, the 3x3 LapXacian operator may be 

used: 



30 




Alternatively, to accommodate the geometric 
characteristics of the CCD raster and to give greater 
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emphasis to the vertical edges, the follo»(/ing kearnel may be 
used: 



-% -1 -% 
-2 8 -2 
-% -1 -% 



In an alternative embodiment, the edge detector block 
56 could be implemented as separate horizontal and vertical 
10 convolution operations (such asalx3or3xi matrix) , 
with additional logic to avoid overemphasis of diagonal 
edges. This alternative embodiment is less hardware 
intensive and gives improved picture quality in some 
circximstances , 

15 Edge suppress block 58 receives the vertical low-pass 

intensity (Y^jp) signals from vertical low-pass filter 52 and 
outputs edge suppression function to edge multiplier 60. 

The edge suppression function varies between 0 and l 
in the long exposure processing block 24 only. In the 

20 short exposure processing block 26, the function is set to 
1, i.e., no edge suppression at this point. The function 
is typically implemented in block 24 in a piecewxse linear 
fashion as follows: 

25 f 1 if ^vip < LOWSAT 

Yvto - LOWSAT 

— if LOWSAT < Y^H. < DEEPSAT 

DEEPSAT-LOWSAT * 

if DEEPSAT ^ Y^^ 

30 Typically LOWSAT is set to approximately 190 and 

DEEPSAT to approximately 220. 

Edge multiplier 60 receives input from blocKs 56, 58 
and generates suppressed edge ed^ to intensity (Y) result 
calculation. 

35 Edge multiplier 60 multiplies the edge outp^it of the 

edge detector block 56 by the edge suppression function 
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from block 58 to generate an output value eci,^ to joint 
operations block 64 (see Figure 1) . The purpose of this 
multiplication is to suppress distorted large edges that 
may appear in the long exposure at intensity (Y) values 
5 near saturation, at the same time as they appear in the 
short exposure at lower values of intensity (Y) • The 
double appearance of such edges was found empirically to 
cause the resulting displayed edges to be overemphasized 
and sometimes smeared on account of blooming in the long 

10 exposure* The long exposure edge is suppressed so that 
only the short exposure edge will pass through to the 
output image. The edge suppress function may also be used 
to reduce the amplitude of edges from the long exposure 
which may be otherwise exaggerated due to the higher gain 

15 of the long exposure relative to the short exposure. 

Additionally, as shown in phantom in Figure 5, an 
optional multiplier or LUT (block 57) may iDe added to 
multiply the output of block 56 times the ratio of exposure 
times (duration of long exposure/duration of short 

20 exposure) or the corresponding gain ratio, or some function 
of the exposure and/or gain ratio- This reflects the ratio 
of scales of these two values. 

In the above manner, Y path block 3 0 outpnts processed 
luminance Y^^, edge, and edge^ to point processing block 62 

25 and joint operations block 64. 

Referring now to Figure 6, one sees that point 
processing block 62 includes four point processing 
functions, all of which receive output intensity (Y^) 
values from the long and short exposure processing blocks 

30 24, 26 (see Figure 1). These functions may typically be 
implemented as LUTs in RAM or ROM memory. Point processing 
block 62 generates arbitrary function values for input to 
the joint operations block 64 (Figure 1) . 
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The four "tables of block 62 are: 

1. The intensity (DC result) block 66 which 

generates a LUT value of intensity (Y^) for the joint 
operations block 64. 
5 Block 66 controls the amount of point ("DC") 

Iviminance that is summed with the edge information in 
generating the output Itiminance/ Y,^^ In its most 
general formulation, 

10 yH« = f(Yr > 

where f is an arbitrary function. It has been found 
that a quasilogarithmic or fractional power dependence 
of Yfcrt. on the inputs gives the best output image 
appearance, and the general function above can 
15 generally be reduced to a more compact LUT or 

piecewise linear implementation. 

One simple possible computation of Y^rt is as 

follows: 

^) Yrfjort is multiplied by the exposure ratio, so that 
20 it is on the same scale as Yy^. That is, if a 

certain pixel x is acquired within the active 
(linear) sensitivity region of both the short and 
long exposures, then Yi«g(x) = ^*Y^(^) , where R 
is the exposure ratio, R=long exposure time/short 
25 exposure time (or any other ratio representing 

the two sensitivities) . 
b) Subsequently, Yi^^ and It*Ya^ are linearly 
combined, so that the sum of their relative 
weights is always 1. That is, Y„A=a*Yu^ + 
30 b*R*YriK«# l>a>0, l^bSrO, and a+b=l (the wdr index 

stands for 'wide dynamic range'). The common 
practice is to set a=l in the region where the 
short exposure is cut-off (too dark) , b=l in the 
region where the long exposure is saturated (too 
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bright) ^ and a>0, b>0 in the region where both 
exposures carry meaningful information. However, 
this does not cover all cases, e.g. when neither 
exposure carries any information (long is 
saturated and short is cut-off, or both 
saturated, or both cut-off) . 
c) Finally, the dynamic range of is reduced 

(yielding Y^} by either a logarithmic function, 
or by multiplying it by a small fraction, or by 
using any empirically found mapping which 
resembles the log function or a similar contrac- 
tion, e.g., the square root, 
other possible values for comprise empirical 

modifications of the function described above. 
15 2, & 3. Color weight normalize blocks 68, 70 for 

long and short exposures, respectively, which 
generate normalizing color weights Wj/Y, and w,/Y,. 
Blocks 68 and 70 control the proportions of 
mixing the color values, dr and db, from the long 
20 and short exposures, respectively, that will be 

used to generate the output color values, dr,^ 
and dbn^. Generally, Wj and are chosen so as 
to give predominant weight at each pixel to the 
color values taken from the exposure in which the 
25 intensity (Y) luminance values are in the linear 

portion of the range, and to give a smooth 
transition over luminance gradient regions of the 
image. For the most part, W| and w, are deter- 
mined on the basis of Yj^(long) alone, except for 
30 cases where the long exposure is near saturation 

while the short is near cutoff, so that neither 
gives a linear reading. 

The weighting values are complementary, 
i.e., w, = 1 - w,, and W|, w, > 0. The outputs of 
35 blocks 68 and 70 are normalized by division by 
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the corresponding values of Y^^ for the long and 
short exposures • Preferably, a floating point 
representation for the output values of blocks 
68, 70 is used so as to maintain sufficient 
accuracy to prevent noticeable quantization in 
the output image. 

Alternatively, instead of weighted addition 
of the normalized color from the two exposures, 
simple selection of the normalized color from one 
exposure or the other, with an ordered dither 
(alternation) of the color selection in areas of 
transition may be used. In this case, when Wj = 
1, the color is selected from the long exposure; 
when w, " 1, it is selected from the long 
exposure, and when neither w factor is 1, color 
values are taken alternately from long and short, 
according to a pseudo-random probability 
distribution in which the long and short color 
value probabilities are equal to the w, and w, 
values, A normalized color value that is an 
average of the long and short values may also be 
mixed into the dither, in ordex to give a 
smoother color transition. 

Saturation color suppression factor block 72 
generates the color suppression factor Wht that 
reduces chroma saturation (adds vhite to the 
image) in areas of luminance saturration of the 
input image. An additional edge color 

suppression factor, 2^, is computed in the joint 
operations block (as will be described herein- 
after) . The minimum of Wht and z^, l^oth of which 
vary from 1 to 0, multiplies the chroma compo- 
nents at the output stage of color conversion. 
Thus, as Wht approaches zero, so does the color 
saturation of the output image. 
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The purpose of the saturation color 
suppression function is to reduce the appearance 
of color artifacts that arise due to CCD 
saturation. The linear relationships between the 
5 a, ^, y, and 6 CCD outputs and the true RGB 

colors break down as the CCD 14 approaches 
saturation. As non-linear deviations cannot be 
readily corrected, suspected distorted colors are 
"whitewashed". Similar techniques are used in 
10 the analog domain in conventional CCD cameras. 

As shown in Figure 6: 

Wht = Wi + w,*z, 

Wj and w, are identical to the above color 
weighting values. The variable Z, is a function 

15 of Yo„t ( short ), varying between 0 and 1, as shown 

schematically in the lower right corner of Figure 
6. It tends to zero in areas where the short 
exposure luminance approaches either saturation 
or cutoff. This function will give Wht - 0 at 

20 the saturation end (where generally w, = 1 while 

Wi - 0) . At the cutoff end, normally W| w 1 as 
long as there is adequate overlap between the 
long and short exposures, so that in this range 
the function will usually give Wht « i. 

25 In normal mode, in which only one input 

channel is operative (see explanation below) , Wht 
~ 1 from y=0 up to the low saturation threshold 
(typically 190) . From this threshold up to the 
deep saturation limit of Y (typically 220) , wht 

3 0 drops linearly to its saturation value of 0. In 

replay mode (see below) , there is no saturation 
color suppression. 

Figure 7 discloses the joint operations block 64 (also 
see Figure 1) . 
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Joint operations block 64 combines the chrominance and 
liaminance data from the long and short exposure processing 
blocks 24, 26, together with data from point processing 
block 62, to generate a combined Y/dr/db result. Block 64 
5 then converts this result to output in standard RGB or 
Y/Cr/Cb (luminance, chrominance (red) and chrominance 
(blue)) color space. A color suppression factor Z is 
computed and applied to the chrominance outputs in order to 
reduce color artifacts (by reducing chroma saturation) 
10 around edges and areas of luminance signal saturation. 

Joint operations block 64 includes: 

1. The dr, db, Y block 74 (recalling that dr 
and db are the differences between 
successive readings in even and odd lines, 

15 respectively) which receives dr, db values 

from the color path outputs of long and 
short exposure processing blocks 24, 26 
respectively; ed^ from tlie intensity (Y) 
path output of long exposure processing 

20 block 24 and edge data from the intensity 

(Y) path output of short exposure processing 
block 26; and Ym^, Wj/Yj and w,/Y, from table 
processing (LOT) block €2. Block 74 
generates combined intensity Y/dr/db results 

25 to color conversion block 78 (to be 

discussed) . Block 74 will be discussed in 
greater detail hereinafter. 

2. The color suppression factor block 76 which 
receives and ed^ froia edge detector 

30 block 56 and saturation color suppression 

factor (Wht) from point processing block 62 
and generates chroma suppression factor Z 
for color conversion block 78. Block 76 
will be discussed in greater detail 

35 hereinafter. 
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3. The color conversion block 78 which receives 
dr^* dbccuit from block 74 and Z, the 
color suppression factor from block 76 and 
generates R«rt, G^, and B^^ and cr and Cb. 
5 Block 78 will be discussed in greater detail 

hereinafter. 

The dr, db, Y block 74 is shown in further detail in 
Figure 8. 

Block 74 includes an intensity (Y) calculation which 

10 is performed by adders 79, 80 and edge limiting block 81. 
Adder 79 receives ed^ (long) data from long exposure 
processing block 24, and ed^ from short exposure 
processing block 26. These two inputs are added to give 
edge^, which is then input to the edge limiting block 81. 

15 Edge limiting is implemented as a piecewise linear function 
with 6 inflection points (A1...A6) and 4 slopes (S,...S4), as 
shown in the upper right inset of Figure e. Generally the 
inflection points and slopes are chosen so as to enhance 
the smaller edges (i.e., and S3 2: 1) , while large edges 

2 0 (edge > A5 or < Aj) are suppressed. Since ^hese large edges 
come through strongly in the Y^n- contribution anyway, the 
output image has a more pleasing appearance if they are not 
additionally enhanced. A3 and A4 may be set: to 0, but it is 
sometimes desirable to set them to small non-zero values in 

25 order to suppress false edges due to noise. The best 
results appear to be obtained with |Ai| and |a<s| values of 
50 to 60. The best values of the slopes \ S,\ are typically 
in the range 0.5 to 2, but the hardware allows a greater 
range . 

30 The edgeu^ output is then summed by adder 80 with the 

Ylut output of block 62 to obtain the output luminance value 
Y^. Additionally, as shown in phantom in Figure 10, adder 
80 may be removed from its location in Figure 8 and placed 
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so thac the output of block 81 is not added to until 
just before being added into block 113^^, that is, as late 
as possible. 

5 Block 74 further includes a dr, db calculation which 

is performed by the remaining sections of block 74, The 
dr, db calculation receives low-pass color components dr, 
db from the color paths of long and short exposure 
processing blocks 24, 26; w,/Y, and w,/Y, from block 62; and 

^0 as calculated by adder 80. The dr, db calculation 

outputs dr^ and db^. 

The long and short values of dr and db are multiplied 
by the respective normalized color weights, W|/Yi and w,/Y, by 
multipliers 82, 84, 86, 88* These normalized, weighted 

15 color values from the two exposures are summed together by 
adders 90, 92 and then multiplied by Y^ by multipliers 94, 
96 to give the scaled values: 

1W| W ) 

<i^i — + dr. 

= * [db, ^ + db. 

I Y, Y, J 

Alternatively, dr^ and db^ may be generated by 
25 selection between the long and short normalized dr and db 
inputs (and possibly their long/ short average values) , 

The color suppression factor block 76 of Figure 7 is 
shown in more detail in Figure 9. 

Maximum value block 100 selects the higher of the two 
30 absolute values of ed^^ and ed^ as calculated by absolute 
value blocks 98, 99. The result of the calculation of 
block 100, ed^, is input to edge chroma suppression factor 
block 102 to calculate Z^j. The calculation of is 
implemented as a piecewise linear function, shown in the 
35 upper right corner of Figure 9. As can be seen in Figure 
9, receives a value between Th and 1, given by: 
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if ed^<El 



E2 - Bdj^ 



Th + 



E2 - El 



(1 " Th) if El<ed^<E2 



Th 



if E2£ed^ 



10 



15 



20 



25 



30 



Typically, El=10 and E2=27 have been found to give 
good results. The minimum value of Z^, Th, is ordinarily 
set to zero, to give complete chroma suppression at very 
strong edges. Th ?^ 0 is used only in replay of images 
stored in mosaic format (see generate mosaic block 120 
described hereinafter) , in which case serves to suppress 
color anomalies resulting from the reinterpolation of the 
pixel values. 

Thereafter, as shown in Figure 9, minimum value block 
104 selects the minimum of the two color suppression 
factors, and Wht, thereby determining the edge ciriterion 
or saturation criterion that should be used to provide the 
required degree of chroma suppression at the given pixel • 

Referring now to Figure 10, which discloses in detail 
color conversion block 78 of Figure 7, one sees that color 
conversion block 78 receives drn^uiir and dh^^ from block 

74 and Z from block 76 and generates outputs in fcjoth the 
RGB and Y/Cr/Cb formulations. 

In other words, block 78 takes the interim dynamic 
range enhancement results Y/dr/db, and converts tirxem into 
conventional color components for system output. 

Block 78 includes horizontal low-pass filter 106 which 
receives Y^ and calculates Y^ (Ip) for the coloir matrix 
block 108. 

Horizontal low-pass filter 106 is identical to the 
low-pass color component block 36 in the color patli block 
28 (see Figures 3 and 4) . Since the dr and db inputs to 
the color matrix 108 have already been low-pass filtered by 
this low-pass filter operator, it is necessary to filter 



SUBSTITUTE SHEET (RULE 26^ 



wo 94/18801 



PCT/US94/01358 



27 

the intensity (Y) value as: well in order to prevent color 
artifacts. 

Color matrix block 108 receives Y,^ (Ip) from 
horizontal low-pass filter 106 and dr^ and db^ firom block 
5 74 and generates low-pass RGB color component outputs. 

If one recalls the derivation of Y, dr and db from the 
original a, ff, y and 6 values of the mosaic CCD input: 



V = g + ^ = g + g 
10 2 2 

dr = 7 - Q£ 

db s 5 - )3 

together with the RGB ec[uivalencies of a, jS, 7 and 6, one 
obtains the following relationships between RGB and the 
15 Y/dr/db values: 









0.2 


0.4 


0.1 




Y 


20 


G 


= 3.5 * 


0.4 


-0.2 


0.2 




dr 


B ^ 




0.2 


-0.1 


-0.4 , 




db 



25 



The factor of 3.5 is required for normalization of the 
relation Y = R-i-1.5G + B. Due to hardware implementation 
considerations r the color conversion matrix is calculated 
as follows: 





R ] 




^1.0 


2.0 


0.5 ] 




Y 




G 


- 0.7 * 


2-0 


-1.0 


1.0 




dr 


30 


B 




1.0 


-0.5 


-2.0 




db [ 



In this way the matrix multiplication is performed by 
a series of shift/add operations. The multiplicative 
factor 0.7 is combined (by multiplication) with externally 
programmed RGB white balance correction factors as is 
35 described hereinafter. 
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RGB white balance multipliers 109^, 109^, 109c teceive 
low-pass RGB signals from color matrix block 108 and 
generate normalized low-pass RGB signals. 

Multipliers 109^, 109^, 109c multiply each of the RGB 
5 low-pass values by a pre-computed white balance corirection 
factor, adjusted by the normalization factor 0.7 r^cjuired 
by the color matrix calculation. Although conventional RGB 
white balancing uses only two multiplicative factors, 
correcting R and B while G is held constant, this "short 

10 cut" does not preserve constant Y achromatic luminance. 
This loss of normalization may lead to the appearance of 
artifacts and incorrect luminance in the output. It is 
necessary, therefore, to use three multiplicative factors, 
normalized to preserve constant luminance Y. 

15 The calculation of the correction factors is performed 

off-line by capturing a white image and selectively 
computing average values R, G, B, and Y, excluding pixels 
near saturation or cutoff. From the definition Y = R + 
1.5G + B, it follows that for a corrected pixel ±n the 

20 white image, it should be found that: 

R=G = B = -5^Y 

From this relationship one derives the correction factors 
25 to be used in multipliers 109^/ 109^, 109c: 

1 Y 

R factor = — = =- 

3.5 R 

30 G factor = — X- 

3.5 G 

1 Y 

B factor = — ^ ~ 

3.5 B 

35 Output signal enhancement block 110 (which includes 

chroma suppression and RGB output functions) receives 



SUBSTITUTE SHEET (RULE 26) 



wo 94/18801 



PCT/tJS94/01358 



29 

corrected low-pass RGB color component signals from color 
matrix block 108 via multipliers 109^* 109b, 109c; J^iewn ^^om 
block 74; Y,^ (Ip) from block 106; and chroma suppression 
factor Z from block 76. 
5 As noted above, the RGB values output from color 

matrix block 108 are low-pass values. High-frequency image 
information is "re-injected" into RGB according to the 
following equation (given here only for the R component, 
since the treatment of G and B is identical): 

Rhp = Rip + K*Y^ - K*Y^^ 

K is an arbitrary constant between 0 and 1, chosen 
according to the degree of high-frequency enhancement 
required. Values in the range 0.4 < K ^ 0.8 are typically 
15 used. 

The addition and subtraction of Y^ values to the RGB 
components can alter the original values of R/G and B/G, 
with the result that the correct hue of the image is not 
preserved. Therefore, in an alternative embodiment, R, G 
20 and B are multiplied by a high -pass enhancement function. 

Since the RGB color component values contain both 
luminance and chrominance information, the chroma 
suppression factor, Z, is best applied to chrominance-only 
components, by adders 113^, 113^, 113c: 



25 


Or = 


R - 






eg = 


G - 


^result 




Cb = 


B - 





Combining these equations with the previous ones for "high 
frequency re-injection", one obtains the following formula, 
30 which is implemented as shown in Figure 10 (including 
arithmetic element blocks 114, 115, 116) to obtain Cr from 
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Cr = R^, + .K*(Y^ - yI^) - 

and likewise for Cg and Cb. These Cr/Cg/Cb values are 
multiplied by Z by multipliers 112^, 112b, 13.2^. At this 
5 point, Y/Cr/Cb output is available directly (using Y = 
Ynauh) / though it is preferable to add a bias of +128 to the 
signed digital outputs Cr and Cb in order to convert them 
to positive values for D/A conversion. In the alternative, 
Y^ can be added back into the chroma-suppressed Cr/Cg/Cb 

10 values (by adders 113a-113c) to obtain the final ^/G^/Bg^A* 
Referring now to Figure 11, which discloses generate 
mosaic block 120 of Figure 1 in more detail, one sees that 
the input of generate mosaic block 120 is Rom/ Gout/Bom from 
color conversion block 78 of joint operations block 64, 

15 The output of block 120 is the equivalent a, 7, S values 
in the format; 

^cq Tcq ^cq Ifcq • • • • • 

^cq ^cq *^cq 

^eq Teq ^cq Teq • • • • ♦ 
20 0^ ^cq ^cq ^cq 

In order to reduce memory requirements for image 
storage and to allow stored images to be replayed through 
the apparatus 10 for display, the final RGB values from the 
processed image are used to generate equivalent:, simulated 
25 mosaic values of at, )S, 7, and S. In this way, only eight 
bits per pixel of information must be stored, rather than 
the 24 bits of full output information. Tliese mosaic 
values can later be replayed to regenerate the stored 
image* 

30 The simulated mosaic values are generated by the 

following matrix in matrix block 122, based on the color 
equivalencies given hereinabove* 
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0 2 1 
112 
2 11 
12 0 



10 



15 



20 



25 



30 



35 



The factor of h that multiplies the matrix is used for 
reasons of hardware convenience — in order to ensure that 
a, jS, y, and 6 do not overflow the range 0-255 of 8 bits* 
To maintain the normalization relations given hereinabove, 
the factor should actually be 1/3 .S, Therefore, in oreplay 
mode, mosaic white balance block 32 is used to multiply the 
&9 7/ and S values back by 4/3.5 (-8/7) iaefore 
reprocessing. Finally, multiplexer 124 selects which one 
of the four mosaic values to output for each pixel 
according to the table: 



Pixel j j 





j even 


j odd 


i even 


CL 


7 


i odd 




S 



Apparatus 10 has three modes of operation: normal, 
adaptive sensitivity (AS) , and replay • 

!• Normal mode emulates the performance of a mosaic color 
CCD camera without adaptive sensitivity. In this mode only 
the long exposure portion of the pipeline operates. The 
processing functions are limited to decoding the mosaic 
input into conventional color components: Y/Cr/Cb oir RGB, 
while additionally performing filtering operations for 
anti-aliasing, detail (edge) enhancement and cJiroma 
suppression where required. 
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2. Adaptive sensitivity mode uses all the resources of 
the processing pipeline to generate wide dynamic range 
images as described hereinabove. 

3. Replay mode is required for displaying images that 
5 have been stored in RAM or disk. Apparatus 10 stores these 

images in a regenerated mosaic format in order to save on 
storage memory requirements. Replay mode is similar to 
normal mode, except that most of the enhancement operations 
are not performed: since the stored data have already been 

10 filtered once, it is for the most part not desirable to 
filter them again. 

The preceding specific embodiments are illustrative of 
the practice of the invention- It is to be understood, 
however, that other expedients known to those skilled in 

15 the art or disclosed herein, may be employed without 
departing from the spirit of the invention or the scope of 
the appended claims. 
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CLAIM S 

What is Claimed is: 

1. A color wide dynamic range video imaging apparatus 
comprising: 

5 sensor means for providing a plurality of color video 

images of a scene at different exposure levels; 

means for dividing each color video image into 
components; and 

means for processing said components of each of said 
10 plurality of video images to produce a combined color video 
image including image information from said components of 
each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 
15 wherein said means for processing includes means for 

calculating point intensity data for said each of said 
plurality of said video images. 

2* The color wide dynamic range video imaging 
20 apparatus of Claim 1/ wherein said means for processing 
includes means for calculating color weighting factors. 

3. The color wide dynamic range video imaging 
apparatus of Claim 2, wherein said means for processing 
25 includes means for calculating saturation color suppression 
factors. 

4 • The color wide dynamic range video imaging 
apparatus of Claim 1, wherein said means for dividing each 
30 color video image into components includes a filter means 
in front of said sensor means and said filter means 
includes filter elements of a plurality of colors, said 
plurality of colors corresponding to said components. 
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5. The color wide dynamic range video imaging 
apparatus of Claim 4, wherein said sensor means includes a 
plurality of pixel sensing elements and 

wherein said filter elements are arranged in a regular 
5 repeating pattern with each filter element in front of a 
single pixel element of said sensing means* 

6. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 

10 includes means for evaluating an intensity of each color 
component ; 

means for evaluating color suppression factors for 
each pixel; and 

means for converting said components into RGB color 
15 space. 

7 . The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 
intensity of each color component communicates with a means 

20 for substituting liaminance values when luminance otherwise 
approaches saturation* 

8* The color wide dynamic range video imaging 
apparatus of Claim 6, wherein said means for evaluating an 
25 intensity of each color component communicates with a means 
for limiting luminance values, 

9. The color wide dynamic range video imaging 
apparatus of Claim 5, wherein said processing means 

30 includes white balancing means which calculates correction 
factors for said components based upon the intensity of 
said color video images, 

10. The color wide dynamic range video imaging 
35 apparatus of Claim 9, wherein white balancing means 
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calculates an average intensity of said color video rmages 
excluding saturated pixels and cut-off pixels. 

11 • The color wide dynamic range video imaging 
5 apparatus of Claim 9, including vertical low-pass filter 
means receiving intensity and false edge suppression means, 
wherein said false edge suppression means receives 
intensity data from said vertical low-pass filter means and 
calculates false edge suppression factors which are reduced 
10 when said intensity data exceeds a pre-selected saturation 
value. 

12. The color wide dynamic range video imaging 
apparatus of Claim 11, wherein said false edge suppression 

15 means calculates said false edge suppression factors for an 
image of a longest of said different exposure levels. 

13. The color wide dynamic range video imaging 
apparatus of Claim 12, further including edge detection 

20 means and wherein an output of said edge detection means is 
multiplied times said false edge suppression fact.ors. 

14 . The color wide dynamic range video imaging 
apparatus of Claim 5, wherein color components are 

25 calculated by multiplying an intensity times a sum of 
products of prior color components and output of said means 
for calculating said color weighting factors. 

15. The color wide dynamic range video imaging 
30 apparatus of Claim 6, including means for calculating white 

balance correction factors for said components converted 
into RGB color space, wherein said white balance correction 
factors for a given respective RGB component are calculated 
by dividing average overall luminance by a multiple of an 
35 average value of said respective given RGB component. 
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16. The color wide dynamic range video imaging 
apparatus of Claim 15, wherein said average value of said 
respective given RGB component is calculated excluding 
pixels which are substantially near saturation or 

5 substantially near cut-off. 

17. The color wide dynamic range video imaging 
apparatus of Claim 6, including means for converting said 
RGB color space components into mosaic color components 

10 corresponding to colors of said filter elements of said 
filter means. 

18 . The color wide dynamic range video imaging 
apparatus of Claim 1, wherein said ineans for processing is 

15 implemented on a single chip. 

19. An imaging apparatus comprising: 

sensor means for providing a plurality of color video 
images of a scene at different exposure levels; 
20 means for dividing each color video image into 

components ; and 

means for processing said components of each of said 
plurality of video images to produce a combined color video 
image including image information from said components of 
25 each of said plurality of color video images by applying 
neighborhood transforms to at least one of said components 
of each of said plurality of video images, 

wherein said means for processing includes means for 
calculating point intensity data for said each of said 
3 0 plurality of said video images, and 

wherein said processing means includes white balancing 
means which calculates correction factors for said 
components based upon the intensity of said color video 
images . 

35 
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20. A color wide dynamic range video processing chip 
comprising: 

means for processing components of each of a plurality 
of video images of a scene at different exposure levels to 
5 produce a combined color video image including image 
information from ^aid components of each of said plurality 
of color video images by applying neighborhood transforms 
to at least one of said components of each of said 
plurality of video images, 
10 wherein said means for processing includes means for 

calculating point intensity data for said each of said 
plurality of said video images. 
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